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Abstract 


Virtual charter schools provide full-time, tuition-free K-12 education through internet-based 
instruction. Although virtual schools offer a personalized learning experience, most research 
suggests these schools are negatively associated with achievement. Few studies account for 
differential rates of student mobility, which may produce biased estimates if mobility is jointly 
associated with virtual school enrollment and subsequent test scores. We evaluate the effects of a 
single, large, anonymous virtual charter school on student achievement using a hybrid of exact 
and nearest-neighbor propensity score matching. Relative to their matched peers, we estimate 
that virtual students produce marginally worse ELA scores and significantly worse math scores 
after one year. When controlling for student mobility during the outcome year, estimates of 
virtual schooling are slightly less negative. These findings may be more reliable indicators of the 
independent effect of virtual schooling if matching on mobility proxies for otherwise 
unobservable negative selection factors. 


Keywords: virtual schools, charter schools, mobility, matching 


We thank Brian Gill, Mark Berends, Dillon Fuchsman, Matthew Lee, and Jessica Goldstein for 
constructive comments on previous versions of this paper. 


VIRTUAL SCHOOLING, STUDENT MOBILITY, AND ACHIEVEMENT 


Introduction 
Covid-19 entirely disrupted brick-and-mortar education for most students in the United States. 
As traditional public schools shifted toward virtual instruction, many commentators concluded 
that virtual education was an ineffective substitute for in-person instruction.! It is too early to 
know the extent of learning loss during the pandemic, but it seems likely that some students 
indeed fell behind. By and large, virtual schooling in 2020 was delivered by public school 
districts with little experience in online education. Not only did teachers and administrators 
contend with a new instructional mechanism, but they also had to grapple with a deadly 
pandemic. Accordingly, conclusions about the efficacy of virtual instruction during Covid-19 
may not be generalizable to virtual schooling as we knew it prior to 2020—or as we will know it 
once the pandemic has been contained. 

Having said that, much of the existing literature on virtual instruction offered by full-time 
online education professionals still suggests a negative association with student achievement. 
Much of this research has been conducted with quasi-experimental designs, since random 
assignment is often unfeasible in the context of virtual schooling. Accordingly, it is unclear 
whether prior studies have adequately controlled for differences between virtual and comparison 
students. Our evaluation of a single, large, anonymous virtual charter school leverages data on 
student mobility, at least in the outcome year of evaluation. We provide evidence that failing to 
account for school transfers may modestly bias the effects of virtual schooling in a negative 


direction. 


' See Gould, E. (2020). “Remote Learning Is a Bad Joke.” The Atlantic. Retrieved from 
https://www.theatlantic.com/ideas/archive/2020/08/kindergartener-virtual-education/615316/; Natanson, H. & 
Meckler, L. (2020). “Remote school is leaving children sad and angry.” The Washington Post. Retrieved from: 
https://www.washingtonpost.com/education/2020/1 1/27/remote-learning-emotional-toll/?arc404=true; and Hobbs, 
T.D. & Hawkins, L. (2020). “The results are in for remote learning: It didn’t work.” The Wall Street Journal. 


Retrieved from: https://www.ws].com/articles/schools-coronavirus-remote-learning-lockdown-tech-11591375078 
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Background on Virtual Schooling 
Virtual schools, also referred to as cyber schools or online schools, provide full-time, tuition-free 
K-12 education to nearly 300,000 students in 35 states (Molner et al., 2019). Although virtual 
students usually receive education from their home residence, virtual schooling is distinct from 
homeschooling. The defining feature of virtual schooling is the web-based delivery mechanism. 
Virtual schools often provide computers and software, and students are connected to instructors 
through the internet, video-conference, and email. Lessons may be synchronous or 
asynchronous, depending on the school and course, allowing students to interact with teachers 
and peers in real-time. As of 2018, one-fifth of virtual students were enrolled in district-managed 
virtual schools (Molner et al., 2019). One of the first district-managed virtual schools was 
established by Florida’s legislature in 1997.” 

Virtual charter schools, one form of virtual schooling, are state-funded, state-regulated 
public schools that educate students through internet-based communication. As of January 2020, 
virtual charter schools were authorized in 27 states.* Virtual charters are required to administer 
annual state assessments like brick-and-mortar charter schools. 

Pazhouh and colleagues (2015) underscore the heterogeneous nature of virtual schools in 
lesson delivery. Students are required to be online at different times, experience a mixture of 
online and in-person discussions, and have varying amounts of teacher interaction. Gill and 
colleagues (2015) document that Black students and students with disabilities enroll in virtual 
schools at similar levels to those found in other public schools, although Hispanic students and 


non-native English speakers are somewhat underrepresented. Beck and colleagues (2014) present 


? Florida Virtual School. “About Florida Virtual School.” Retrieved from: https://www.flvs.net/about/flvs 
3 Education Commission of the States. (2020). “Charter Schools: Does state law explicitly allow virtual charter 
schools?” Retrieved from: https://www.crpe.org/sites/default/files/crpe-policy-framework-online-charter-schools- 


final_0.pdf 
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evidence that virtual schools serve disproportionately high numbers of students with emotional 
or academic problems relative to traditional public schools. 

Advocates argue that virtual schools reimagine education through an innovative 
approach, personalized to the unique needs of each student. Virtual schooling has potential to 
offer instruction for children in remote areas who lack access to certain coursework (Heppen et 
al., 2011). In addition, virtual schools can hire teachers from anywhere in each state, 
accommodate teachers who prefer non-traditional work schedules, and employ high teacher- 
student ratios without sacrificing personalized attention for students (Pazhouh et al., 2015). 
Virtual schools are also attractive for students with rare illnesses or unusual travel schedules. 

Parents may pursue non-traditional educational models for various reasons, including 
academic quality, civic or religious values, concerns about safety, curriculum, class size, 
extracurriculars, school facilities, and longer school hours, among others. School choice theory 
suggests student achievement will improve when parents exercise increased autonomy over the 
education of their children (Chubb & Moe, 1990). 

Nonetheless, virtual charter schools are controversial. Critics point to virtual schools’ 
poor track record of increasing achievement relative to traditional public schools. Researchers 
question whether “the poor performance of virtual charter schools reflects a disadvantage 
inherent to online instruction, the unique dysfunction of this particular sector of schools, or some 


combination of the two factors.” In several states, a combination of unions,” school board 


4 Fitzpatrick, B. R., Berends, M., Ferrare, J.J., & Waddington, J. (2020). “Virtual charter schools and online learning 
during COVID-19.” Retrieved from Brookings Institution: https://www.brookings.edu/blog/brown-center- 
chalkboard/2020/06/02/virtual-charter-schools-and-online-learning-during-covid-19/ 

5 “Oregon’s Coronavirus education lockdown.” (2020). Retrieved from Wall Street Journal: 
https://www.wsj.com/articles/oregons-coronavirus-education-lockdown-1 1585697080 
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organizations,° and district leaders’ have called for virtual school moratoriums, enrollment limits, 
and spending cuts. But it is not just opponents of school choice who oppose virtual schooling. In 
2016, three national organizations that support brick-and-mortar charter schools issued a report 


highlighting the need to improve the quality of virtual charter schools.® 


Virtual Schooling Literature Suggests Negative Association with Achievement 
There have been few rigorous evaluations comparing outcomes of virtual students to their 
traditional brick-and-mortar public school peers (Barbour & Reeves, 2009; U.S. Department of 
Education, 2010). Even fewer studies account for the type of students who select into virtual 
schooling. Virtual students may be, on average, negatively selected based on prior test scores, 
disability status, and poor experience in a previous school. A virtual student may also be 
positively selected if he or she is unchallenged academically in traditional schools. 

Existing evaluations conclude that virtual schooling has significant and large negative 
effects on student achievement, especially in math. None of the studies of virtual schooling have 
been experimental, since online instruction can be provided to most students who desire it. It is 
not clear if the quasi-experimental studies to date have adequately controlled for observable and 
unobservable factors that might bias estimates of virtual schooling effects. 

A research team with the Center for Research on Education Outcomes at Stanford 


University systematically analyzed the effects of virtual charter schools on student achievement 


6 “PSBA supports governor taking steps to address charter funding issues.” (2019). Retrieved from Pennsylvania 
School Boards Association: https://www.psba.org/2019/08/psba-supports-governor-taking-steps-to-address-charter- 
funding-issues/ 

7 Mezzacappa, D. (2020). “District leaders call for moratorium on new charters until law is changed.” Retrieved 
from Philadelphia Public School Notebook: https://thenotebook.org/articles/2020/01/27/district-leaders-call-for- 
moratorium-on-new-charters-until-law-is-changed/ 

8 “A call to action to improve the quality of full-time virtual charter public schools.” (2016). Retrieved from 
National Association of Public Charter Schools: https://www.publiccharters.org/sites/default/files/migrated/wp- 


content/uploads/2016/06/Virtuals-FINAL-06202016-1.pdf 
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in multiple states (Woodworth et al., 2015). Researchers collaborated with 18 state departments 
of education to obtain data on virtual charter and traditional public school students. For each 
observed virtual charter student, a match was generated by drawing on the records of public 
school students with identical traits and similar prior achievement. Although the researchers 
provided descriptive statistics on student mobility, they did not include mobility as a control 
variable in their statistical models. The study found large negative results for virtual charter 
students compared to peers in traditional public schools. Virtual charter students produced 25 
percent of a standard deviation lower gains in math and 10 percent of a standard deviation lower 
gains in English language arts (ELA). 

Other quasi-experimental research also found negative associations between virtual 
charter attendance and student achievement. For example, Fitzpatrick and colleagues (2020) used 
longitudinal student records in Indiana to compare students who switch from traditional public 
schools to virtual charters against matched peers with similar observable characteristics. The 
authors found that virtual students perform one-third of a standard deviation worse in ELA and 
one-half of a standard deviation worse in math compared to their peers. Ahn and McEachin 
(2017) found virtual students in Ohio perform 40 percent of a standard deviation worse than 
comparable students in math and 20 percent of a standard deviation worse in ELA. Additionally, 
Bueno (2020) found that virtual students in Georgia demonstrated achievement losses ranging 
from 10 percent to 40 percent of a standard deviation in multiple subjects. An evaluation of a 
virtual charter school in a southern state, using a matching design, found that participating 
students experienced statistically significant negative achievement effects in ELA and math 


during their first three years of virtual schooling (Lueken et al., 2015). 
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Our interest is in public charter schools that deliver instruction through a virtual modality. 
The intervention is a composite of charter schooling and virtual schooling. A separate research 
literature exists on virtual schooling delivered by traditional public schools (e.g., Evergreen 
Education Group, 2017; Schwerdt & Chingos, 2015). As has been made clear during the Covid- 
19 crisis, charter schools and traditional public schools deliver virtual schooling differently, 
whether under planned or unplanned circumstances (Kingsbury, 2020; Vanourek, 2020). 
Students also vary in the extent to which they use computer technology at home to complete 
schoolwork, with some studies indicating more home computer use is associated with lower 
student achievement (Agasisti, Gil-Izquierdo & Han, 2020; Fairlie, Beltran & Das, 2010). The 
general research literature on education technology is vast and variegated. Thus, we focus on the 
narrower research literature specifically on virtual charter schooling. 

Student Mobility: A Predictor of Low Achievement 

A student may be considered mobile if he or she transfers from one school to another for reasons 
other than a grade promotion. In our study, student mobility is measured by counting the number 
of transfers that occur during a school year. School transfers during the academic year are likely 
to harm student learning since they interrupt the sequencing of concepts and skills. Mobility, also 
referred to as churn or transience, can occur for many reasons. Families may seek academic 
programming at a different school, a parent may assume a new job in a new location, or the 
student may suffer from bullying or dissatisfaction in the previous school (Sparks, 2016). 

A meta-analysis by Mehana and Reynolds (2004) found student mobility was a strong, 
negative predictor of achievement. Similarly, Hanushek & colleagues (2004) estimated that 
student learning losses, particularly those experienced in the first year, had spillover effects on 


other students. Kerbow (1996) found that high rates of student mobility had large negative 
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implications for schools in addition to the students themselves. South and colleagues (2007) used 
the National Longitudinal Study of Adolescent Health to estimate that mobile students were 
twice as likely as non-mobile peers to drop out of school, although mobile students differed on 
observable variables that may have influenced both mobility and persistence. 

Virtual students are highly mobile. Gill and colleagues (2015) found that the average 
student in a virtual school enrolled for only two years. As a result, some researchers studying 
virtual schools argue that student mobility should be included in statistical models that compare 


achievement of virtual students to traditional public school students (Gatti, 2018). 


Data 

We study a full-time, accredited virtual charter school. Any K-12 student residing within the 
anonymous state is eligible to attend the school, tuition-free. Students receive synchronous and 
non-synchronous instruction, complete state assessments, and are eligible to participate in 
extracurricular activities with their resident school district. The school employs full-time, 
licensed counselors to assist with personal development as well as college and career planning. 

We use statewide student-level achievement and demographic data from the 2014-15 
through 2017-18 school years. The outcomes of interest are ELA and math scores, standardized 
by year and grade. The data include indicators for gender and race, as well as free- or reduced- 
price lunch (FRPL) eligibility, English as a Second Language (ESL) status, and special education 
(SPED) status. 

Treatment is defined as being enrolled in the virtual school of interest at any time during 
the outcome year. The comparison condition is defined as not being enrolled in the virtual school 


of interest during the outcome year. We observe if a student transferred to a different school 
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during the school year but not specifically when the transfer occurred. Thus, we code any student 
exposed to the virtual charter school during the outcome year as “treated.” 

This school educates a non-trivial percentage of the nation’s virtual sector. Its large 
enrollment permits a well-powered analysis and increases the salience of our research. Our data 
include 12,498 unique students in the virtual school and 1,061,165 other public school students 
with whom the virtual students may be compared. We limit the sample to grades 3 through 8, 
reducing the analytic sample to 6,054 virtual students and 574,633 potential comparison students 
since these are the grades assessed by the state. Then, because students must have baseline and 
outcome test scores to be included in our analysis, the sample is further reduced to slightly more 
than 3,500 virtual students and slightly less than 400,000 comparison students (Table 1). Student 
counts in ELA are slightly different from student counts in math because of variation in missing 


test score data, explained in more detail below. 


Table 1: Unique Students Who Meet Criteria for Inclusion in Analytic Samples 


ELA Math 

Virtual Comparison | Virtual Comparison 

Total Unique Students 12,498 1,061,165 | 12,498 1,061,165 
% of total unique students | 100% 100% | 100% 100% 
In Grades 3-8 6,054 574,633 | 6,054 574,633 
% of total unique students 48% 54% 48% 54% 
With baseline test scores 3,888 406,177 | 3,890 407,106 
% of total unique students 31% 38% 31% 38% 
With outcome test scores 3,505 393,632 | 3,509 393,824 
% of total unique students 28% 37% 28% 37% 
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To quantify mobility, we create a variable identifying each student’s number of mid-year 
school switches during the outcome year. For example, a student who remains in the same school 
for an entire school year is coded as “0” mid-year switches, while a student who changes schools 
twice during one school year is coded with “2.” Less disruptive transfers between school years— 
over the summer, for example—are not captured by this variable. 

Table 2 presents baseline descriptive statistics for our analytic sample. Students in the 
virtual school have statistically significant and practically meaningful observable differences 
from other public school students. Virtual students demonstrate higher lagged ELA scores and 
lower lagged math scores than their peers. Unsurprisingly, the average virtual student has more 
mid-year school transfers than his or her peers. Virtual students are substantially more likely to 
be white and less likely to be Black. Students in the virtual school are less likely to be identified 
with ESL, SPED, or FRPL status. 

Since the virtual school does not operate a traditional school lunch program, it is possible 
some low-income families did not complete the necessary paperwork for their child to be 
deemed eligible for the program. However, FRPL eligibility can be the basis for other benefits, 
such as receiving free education technology. It is therefore unclear whether parents of students in 
the virtual school are less likely to submit FRPL paperwork. Absent clear evidence to the 
contrary, we assume that FRPL rates are not systematically biased across the treatment and 


comparison conditions, though we acknowledge such bias is possible. 
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Table 2: Descriptive Baseline Statistics (2015-16) 


Virtual Comparison Difference 


Lagged ELA (normed) 0.09 0.00 0.09° -*#* 
Lagged Math (normed) -0.14 0.00 -0.14 *** 
Mid-year School Switches 0.35 0.06 O29, 
Female 052 0.49 OL03. <* 
FRPL 0.57 0.60 -0.03 ** 
ESL 0.01 0.05 -0.04 *** 
Special Education 0.12 0.12 0.00 
White 0.74 0.52 O22 ste" 
Black 0.12 0.34 20.22. *** 
Hispanic 0.06 0.08 -0.02  *** 
Asian 0.01 0.02 -0.01 


n= 1,532 n=267,658 


Notes: The number of virtual students, 1,532, is less than the number of virtual students in Table 3A and 3B because 
Table 3A and 3B include virtual students from multiple outcome years, not just 2015-16. The 2015-16 school year is 
the first year for which outcome data can be observed, because 2014-15 serves as the baseline measure of the 
outcome variable. 

*** nD < 0.01, ** p< 0.05, * p<0.1 


Missing Data 
In administrative records with over one million students spanning multiple years, missing data 
are to be expected. Fortunately, missingness is relatively rare among both dependent and 
independent variables. Less than five percent of ELA or math scores in the entire sample are 
missing. Conditional on observable characteristics, virtual students are six percentage points 


more likely than potential comparison group students to report missing achievement data, in both 
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subjects. Given the overall low rate of missingness, this differential attrition is consistent with 
the quality standards of the What Works Clearinghouse.’ 

Less than one half of one percent of the values of independent variables such as race, 
gender, FRPL, and ESL indicators are missing. We used multiple imputation by chained 
equations to address these missing cells within our matrix of independent variables. The results 
we present include imputed values for the independent variables. All points estimates are robust 
to the sample without imputed data, which is plausible given the overall low rates of 
missingness. 

Empirical Strategy 
The ideal research design to evaluate the virtual school would be an experiment with random 
assignment. Unlike brick-and-mortar charter schools constrained by limits on physical space, 
virtual schools typically can accommodate all students who seek to enroll, so random lotteries 
are less likely to occur. The anonymous school in our analysis does not utilize lotteries to 
determine enrollment. 

Given that random assignment is not feasible, a credible analysis of student achievement 
hinges on the strategy for reducing selection bias. As is evident from the descriptive statistics in 
Table 2, the average virtual student differs from students in other public schools. Accordingly, it 
is likely virtual students and their peers differ on unobservable characteristics influencing both 
selection into virtual schooling and future achievement. 

To reduce selection bias, we use a hybrid of exact and nearest-neighbor propensity score 
matching to match students by grade-year cohort, prior achievement, and other demographic 
characteristics (Rosenbaum & Rubin, 1983). These variables control for observable factors that 


° U.S. Department of Education, Institute for Education Sciences, What Works Clearinghouse, Standards Handbook, 
Version 4.1, pg. 12. 
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might otherwise confound estimation of the effects of the virtual school and serve as proxies for 
unobservable differences between the average treated and comparison student. Our matching 
protocol generates a comparison group equal to the number of virtual students in the analytic 
sample. One-to-one matching produces an analytic sample sufficiently sized to generate a well- 
powered analysis. Matching rarely eliminates omitted variables bias, but it can reduce such bias 
to trivial levels (Bifulco, 2012). 

Our analysis includes four components. The first evaluates one-year outcomes of virtual 
students relative to a matched comparison group in other public schools. In this specification, we 
do not control for student mobility. Second, we explore heterogeneous one-year effects based on 
gender, race, FRPL status, and school level. Third, to investigate the role of student mobility, we 
include a mobility variable in our matching protocol and compare these estimates to those 
generated from the specification that did not match on mobility. We hypothesize that failing to 
account for student mobility will downwardly bias the estimate of virtual charter schooling. Our 
assumption, grounded in theory and research on school mobility, is that students with greater 
numbers of school transfers will be negatively selective, on average, compared to other students. 
Fourth, we analyze outcomes from a restricted sample of students who do not transfer schools in 
the outcome year. 

Generating Comparison Group through Matching 


We use logit models to predict the likelihood that a student enters the virtual charter school: 


Pr (Di) = 50+ d1Ait-1 + b2Mit + Xitd3 + it 
Our models compare students within cohort cells, ensuring that they are matched to peers 
in the same grade and year. The probability of attending the virtual school (D) is a function of 


prior year test scores in ELA and math (Aiz-7), and student-level covariates (X) including gender, 
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race, FRPL status, ESL status, and SPED status. We exact-match on a band of baseline 
proficiency, gender, and race to limit observable differences on these crucial factors. When 
exploring the role of student mobility, we exact-match on an ordinal variable (M) indicating the 
number of mid-year school switches for each student.!° 

To limit differences in baseline achievement, virtual students are matched with non- 
virtual students who scored within a 5-percentage point band on the prior year’s state assessment 
in the subject of the relevant dependent variable. Students are within-band matched on lagged 
ELA scores and lagged math scores separately. Thus, different comparison groups are 
constructed for math and ELA, respectively. Separate comparison groups are necessary because 
there are not enough comparison students who share similar lagged test scores in both subjects. 
In our preferred specification, we control for prior year performance in both subjects. 

After limiting the pool of potential comparison students to those with the same cohort, 
baseline proficiency, gender, and race, students are matched with their nearest neighbor based on 
propensity scores. We use a caliper of 0.10 and match without replacement.!' The protocol 
generates one comparison student for each virtual student in the final analytic sample. For each 
student pair, the outcome year is the second year that a student appears in the data, with the first 
appearance in the data serving as the baseline year. 


Once the comparison group is identified, we estimate the effects of the virtual school on 


student achievement in a multiple regression framework, using the model: 


'0 Students with two or more mid-year transfers are matched to other students with two or more transfers, but the 
exact number of transfers may differ. In other words, a virtual student with three transfers may be matched to a 
student with two transfers. 

'! Ideally, we would identify a comparison student who is identical on observable characteristics for each virtual 
student in the analytic sample. In practice, there is a trade-off between match quality and the number of virtual 
students included in the analysis. Virtual students are dropped on the rare occasion when no suitable comparison 
students exist who meet the exact-matching and propensity score criteria. In the pages that follow, we report the 
percentage of virtual students who dropped from each matching protocol. 
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Achievementit = a+ fDi + frAir-i+ Xinfs + Eit 

Standardized test scores in ELA or math (Achievementit) for student i in year ¢ are a 
function of an indicator variable for virtual school treatment (D); prior year achievement in both 
subjects (Air-7); and a vector (X) of student-level covariates, including gender, race, FRPL, ESL, 
and special education status; and an error term (€). The coefficient of interest is Ai. 

Results 
One-Year Outcomes 
Tables 3A and 3B present observable differences, after matching, between virtual students and 
their peers. The sample size is so large, at nearly 7,000 students, that even small differences 
between the student groups are statistically significant. However, many of the differences that 
existed between the groups in Table 2 are eliminated after matching. 

Note that in the ELA outcome sample, virtual students have similar prior ELA scores but 
worse prior year math scores. We prioritize baseline equivalence in the subject of the relevant 
dependent variable. The observable differences in the math outcome sample (Table 3B) are 
similar to those found in the ELA outcome sample (Table 3A) with the exception of lagged test 
scores. In Table 3B, the groups are indistinguishable on prior year math performance, but the 
virtual students remain positively selected on baseline ELA scores.'* In both analytic samples, 
virtual students are marginally more likely to have FRPL status and marginally less likely to 


have ESL status. 


'2 Fewer than two percent of virtual students from the final analytic sample are dropped from this matching protocol. 
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Table 3A: Observable Differences After Matching (ELA Outcome) 


Virtual Comparison Difference 
Lagged ELA (normed) 0.08 0.08 0.00 
Lagged Math (normed) -0.08 0.11 a0. 19. Fe 
Female 0.53 0.53 0.00 
FRPL 0.58 0.52 0.06 *** 
ESL 0.01 0.04 -0.03 *** 
Special Education 0.08 0.08 0.00 
White 0.73 0.73 0.00 
Black 0.15 0.15 0.00 
Hispanic 0.07 0.07 0.00 
Asian 0.01 0.01 0.00 


n=3,467 n=3,467 


Notes: See appendix for student counts by grade and year. 
*#  < 0.01, ** p < 0.05, * p<0.1 
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Table 3B: Observable Differences After Matching (Math Outcome) 


Virtual Comparison Difference 


Lagged ELA (normed) 0.07 -0.02 V.09)) = ** 
Lagged Math (normed) -0.08 -0.08 0.00 
Female 0.53 0.53 0.00 
FRPL 0.58 0.55 0.04 *** 
ESL 0.01 0.04 -0.03 *** 
Special Education 0.09 0.09 0.00 
White 0.73 0.73 0.00 
Black 0.15 0.15 0.00 
Hispanic 0.07 0.07 0.00 
Asian 0.01 0.01 0.00 
n=3,473 n=3,473 


Notes: See appendix for student counts by grade and year. 


#7 D < 0.01, ** p < 0.05, * p<0.1 


Table 4 displays achievement differences among virtual and comparison students. In the 


preferred specification, we control for all observable characteristics, including prior year test 


scores. Virtual students perform 4 percent of a standard deviation worse in ELA and 21 percent 


of a standard deviation worse in math than their matched peers. These point estimates are robust 


to a simple specification that only includes an indicator variable for virtual charter schooling. We 


are particularly interested in whether these estimates differ from our mobility-matched estimate, 


which is presented later in Table 7. 


17 


VIRTUAL SCHOOLING, STUDENT MOBILITY, AND ACHIEVEMENT 


Table 4: Virtual School, One-Year Outcomes 


ELA Math 
Simple Preferred Simple Preferred 


Virtual -0.08*** “U.04t** || eO1LOFeS> 020.2148" 
(0.02) (0.01) (0.02) (0.01) 
Lagged Math (normed) Qa Ler O31 
(0.01) (0.01) 
Lagged ELA (normed) 0.614** O2Tt* 
(0.01) (0.01) 
Black -0.05*** -0.11*** 
(0.02) (0.02) 
Hispanic 0.04 -0.02 
(0.03) (0.03) 
Asian 0.10* O.AoFr* 
(0.06) (0.07) 
Mix 0.01 -0.05 
(0.03) (0.04) 
ESL -0.18*** -0.06 
(0.04) (0.04) 
Female O12t#* -0.08*** 
(0.01) (0.01) 
Special Education =), 25*** -0.157** 
(0.03) (0.03) 
FRPL -0.11*** 20,11 *** 
(0.01) (0.01) 
n= 6,934 6,934 6,946 6,946 


Notes: Heteroskedastic-robust standard errors in parenthesis. 
##% 5<0.01,** p<0,05,’* p<0.1 


Heterogeneous Effects 
Heterogeneous effects of the virtual school are presented in Table 5. Negative results in math are 
consistent across gender, race, FRPL status, and school level. In ELA, however, we estimate 
statistically significant heterogeneous effects for females relative to males and for elementary 
schoolers (grades 3-5) relative to middle schoolers (grades 6-8). The estimate of virtual 


schooling on ELA for females is 6 percent of a standard deviation more negative than for males. 
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We speculate that females may be, on average, more social than males and less naturally inclined 
to thrive in the independent environment of online education. The estimate of virtual charter 
schooling on ELA for elementary schoolers is 7 percent of a standard deviation more negative 
than for middle schoolers. We speculate that middle school students are more comfortable using 


computers and maintaining concentration for long periods than elementary school students. 


Table 5: Heterogeneous Effects 


ELA Math 

Female* Virtual -0.06** -0.04 
(0.03) (0.03) 

White* Virtual -0.02 -0.03 
(0.03) (0.03) 

FRPL* Virtual 0.02 0.03 


(0.03) (0.03) 
Elementary*Virtual -0.07** -0.02 
(0.03) (0.03) 


Notes: Heteroskedastic-robust standard errors in parenthesis. Control variables include lagged test scores, race, 
gender, ESL, FRPL, and SPED status. Each point estimate comes from a separate regression. 
*** H<0.01, ** p<0.05, * p<0.1 


One-Year Outcomes, Controlling for Mobility 
The third analysis uses the same matching protocol to generate a comparison group for the 
virtual students, with one exception: we include mobility in the matching process and as a 
control variable.'? We match on school transfers that occur during the outcome year. Matching 


on pre-program mobility would be superior to our approach here, as some unknown 


'3 These sample sizes are marginally smaller than those presented in Tables 3A and 3B because including mobility 
in the matching protocol causes more virtual students to be dropped from the analysis. Fewer than three percent of 
virtual students from the final analytic sample are dropped from this matching protocol. 
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component of outcome-year mobility likely is induced by the treatment of virtual schooling. 
Unfortunately, we cannot reliably determine prior-year student mobility from the data. 

Tables 6A and 6B present observable differences between the samples of virtual students 
and matched peers. The groups are nearly identical on prior test scores in the subject of the 
dependent variable, gender, and race. Virtual students are less likely to be identified as FRPL and 
ESL and are marginally more likely to have SPED status. After matching, virtual and 


comparison students have similar mobility rates. 


Table 6A: Observable Differences After Matching, With Mobility (ELA Outcome) 


Virtual Comparison Difference 


Lagged ELA (normed) 0.07 0.07 0.00 
Lagged Math (normed) -0.09 0.01 -0.10 *** 
Mid-year School Switches 0.51 0.51 0.00 
Female 0.53 0.53 0.00 
FRPL 0.58 0.65 -0.08 *** 
ESL 0.01 0.04 -0.03  *** 
Special Education 0.09 0.08 0.01% 
White 0.73 0.73 0.00 
Black 0.15 0.15 0.00 
Hispanic 0.06 0.06 0.00 
Asian 0.01 0.01 0.00 


n= 3,412 n= 3,412 


Notes: See appendix for student counts by grade and year. 
# D < 0.01, ** p < 0.05, * p<0.1 
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Table 6B: Observable Differences After Matching, With Mobility (Math Outcome) 


Virtual Comparison Difference 


Lag ELA (normed) 0.07 -0.06 Q.13 (oes 
Lag Math (normed) -0.09 -0.09 0.00 
Mid-year School Switches OS 0.51 0.00 
Female 0.53 0.53 0.00 
FRPL 0.58 0.67 -0.08 *** 
ESL 0.01 0.04 -0.03  *** 
Special Education 0.08 0.08 0.00 
White 0.73 0.73 0.00 
Black 0.15 0.15 0.00 
Hispanic 0.06 0.06 0.00 
Asian 0.01 0.01 0.00 


n= 3,390 n= 3,390 


Notes: See appendix for student counts by grade and year. The number of virtual students is smaller for math 
outcomes than ELA outcomes because there are more missing outcome data for math than ELA. 
**E Dy < 0.01, ** p < 0.05, * p< 0.1 


Table 7 compares ELA and math achievement of virtual students after one year against 
their matched comparison group. Notably, the estimated effects on ELA are indistinguishable 
between the groups. In math, virtual students’ achievement lags their matched peers by 18 
percent of a standard deviation. These point estimates are robust to a simple specification that 


only includes an indicator variable for virtual charter schooling. 
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Table 7: Virtual School, One-Year Outcomes with Mobility 


ELA Math 
Simple Preferred _ Simple Preferred 
Virtual -0.01 0.00 <Q, 14***  ..-0.187** 
(0.02) (0.01) (0.02) (0.01) 
Lagged Math 0.20*** 0.50*** 
(0.01) (0.01) 
Lagged ELA OOF F* 0.28*** 
(0.01) (0.01) 
Mid-year Switches -0.06*** -0.06*** 
(0.01) (0.01) 
Black -0.07*** -0.10*** 
(0.02) (0.02) 
Hispanic 0.03 -0.01 
(0.03) (0.03) 
Asian OA O18** 
(0.06) (0.08) 
ESL -0.11** -0.01 
(0.05) (0.04) 
Female 12s -0.09*** 
(0.01) (0.01) 
Special Education -0.29*** Us beads 
(0.03) (0.03) 
FRPL -0.10*** -0.09%** 
(0.02) (0.02) 
n= 6,824 6,824 6,780 6,780 


Notes: Heteroskedastic-robust standard errors in parentheses. 
*** D<0.01, ** p<0.05, * p<0.1 


The findings in Table 7, which account for outcome-year mobility, are 4 percentage 
points of a standard deviation less negative in ELA and 3 percentage points of a standard 
deviation less negative in math relative to the findings in Table 4, which do not account for 
mobility. The ELA estimate in Table 7 is statistically significantly different than the ELA 


estimate in Table 4 at the 99 percent confidence level. The math estimate in Table 8 is 
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statistically significantly different than the math estimate in Table 4 at the 95 percent confidence 
level. Thus, failing to account for mobility may modestly attenuate the point estimate of virtual 
charter schooling.'* While the bias appears downward, it is smaller in magnitude than we 
hypothesized. Still, these results may demonstrate the importance of controlling for differential 
rates of school mobility. Had we not accounted for this difference in our primary analysis, we 
would have falsely concluded that the virtual school had a significantly negative effect on ELA 


outcomes when the true effect may be null. 


Limiting Sample to Non-Switchers 
In the previous section, treated students are matched to comparison students who had the same 
number of school transfers during the outcome year. In both the ELA and math samples, roughly 
48 percent of virtual students received instruction in both the virtual school and another public 
school during the outcome year, which could obscure the treatment effect. To address this 
concern, we conduct a separate analysis restricted to the 52 percent of students who did not 
change schools during the outcome year. These estimates, presented in Table 8, explore 


something akin to a dosage analysis of an entire year of virtual schooling. 


'4 The point estimates in Table 4 and Table 7 are obtained from different samples. We conduct significance tests 
between the estimates by merging both samples, eliminating duplicate student observations, and running our 
preferred specification with separate indicator variables for comparison students in each of the two samples. The 
regression coefficients for each comparison group indicator have a variance and a covariance because they are 
estimated in the same equation, allowing us to test for statistical significance between the two estimates. 
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Table 8: Virtual School Outcomes, Students Without Switches 


ELA Math 
Simple Preferred _ Simple Preferred 
Virtual -0.04 0.00 -0.16*** = -0.21*** 
(0.03) (0.02) (0.03) (0.02) 
Lagged Math Q.22*4* 0.52*** 
(0.01) (0.02) 
Lagged ELA 0.58*** 0.298% 
(0.01) (0.02) 
Black -0.06** -0.09%*** 
(0.03) (0.03) 
Hispanic 0.06 -0.03 
(0.04) (0.04) 
Asian OL 7et* 0.18** 
(0.06) (0.09) 
ESL 201075 -0.06 
(0.06) (0.06) 
Female 12s -0.10*** 
(0.02) (0.02) 
Special Education -0.28*** -0.12*** 
(0.03) (0.03) 
FRPL -0.10*** -0.09%*** 
(0.02) (0.02) 
n= 3,534 3,534 3,506 3,506 


Notes: See appendix for student counts by grade and year. Heteroskedastic-robust standard errors in parenthesis. 
Limiting the sample to students without switches in the outcome year produces differences in observable 
characteristics much like those presented in Tables 6A and 6B, with one exception: differences in lag test scores in 
the non-dependent variable subject are exacerbated. In the ELA outcome sample, virtual students are more 
negatively selected with respect to prior year Math test scores. In the Math outcome sample, virtual students are 
more positively selected with respect to prior year ELA scores. 

*** H<0.01, ** p<0.05, * p<0.1 


Results that exclude switchers are consistent with the findings from the previous section. 
In ELA, the virtual schooling estimate remains nearly zero. In math, the estimated negative 
effect of virtual schooling is 21 percent of a standard deviation, which is 3 percentage points of a 


standard deviation more negative in magnitude than the math estimate in Table 7. 
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Limitations 
We acknowledge this analysis relies on non-random selection of students into the virtual school 
of interest. We think such selection bias, endemic to virtual schooling research, is mitigated by 
identifying comparison students with the same cohort, race, gender, and prior achievement as 
treated students. Although we cannot make causal claims about the impact of virtual charter 
schooling on achievement, we argue that controlling for student mobility in the outcome year 
represents an improvement on other quasi-experimental or observational studies of virtual 
schooling. 

Our study is limited by the fact that we cannot reliably determine prior-year student 
mobility. Some component of our mobility measure might be a product of virtual schooling. We 
attempt to mitigate this obstacle by employing a sample restriction to students who do not 
change schools during the outcome year and find similar results employing both the restricted 
and unrestricted one-year samples. 

Absent random assignment, which is rare in virtual schooling, finding an appropriate 
comparison group for virtual students remains a challenge. Two otherwise identical students who 
only differ in virtual school enrollment are likely to have meaningful unobservable differences. 
Moreover, we do not know why students in our study change schools. Different types of mobility 
likely have different impacts on achievement. For example, a transfer due to a parent getting 
promoted to a new job is not the same as a child transferring because of bullying. Future research 


should explore those mobility-related issues. 
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Conclusion 
Our findings cannot be generalized to school years interrupted by a global pandemic, but we 
hope to shed light on the heterogeneous nature of virtual charter schooling as well as the role of 
student mobility. Although most existing research finds virtual schools have deeply negative 
achievement effects, our analysis of a single, large, anonymous virtual charter school suggests 
that different virtual schools vary in their influence on student learning growth. We estimate 
students in this virtual charter school demonstrate slightly less ELA growth than similar students 
in other public schools. Estimated math effects, while still quite negative, are less negative in 
magnitude than found in other research. We cannot say for sure why our achievement findings 
are less negative than those from most prior studies of virtual schooling. It is possible that this 
school implements an effective combination of personalized learning and rigorous curriculum 
relative to other virtual charter schools. It also may have more experience delivering virtual 
schooling than the virtual providers in prior studies. 

The effects of this virtual charter school on student achievement are not monolithic. 
Students who are female or younger experience more negative virtual schooling effects on ELA 
than their virtually-schooled classmates who are male or older. Our heterogeneous effects 
suggest that virtual charter schooling may be somewhat more effective for boys and older 
students, at least regarding ELA growth. 

We provide evidence that omitting controls for student mobility may downwardly bias 
the effects of virtual schooling on student achievement. Granted, part of the inherent promise of 
virtual schooling is the ability of such schools to capably transition students into the world of 
online education. Regardless of whether virtual schools are fulfilling this promise, however, 


measuring a highly-mobile virtual student against a less-mobile non-virtual student does not 
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produce an apples-to-apples comparison. We suggest researchers account for mobility when 


studying virtual schools, especially using pre-program measures if possible. 
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The following tables identify the grade and outcome year for students included in each unique 


Appendix 


analytic sample. All grade by year cells include an equal number of virtual students and matched 
comparison peers in other public schools. Although students are only assessed in grades 3-8, and 
students must have a lagged test score to be included in the analysis, a third grade student can be 


included in the samples below if he or she is repeating third grade and has a baseline test score 
from the previous year. 


ELA (No Mobility Control) Cohort, Table 3A 


Grade 3 Grade4 Grade5 Grade6 Grade7 Grade 8 Total 

2016 10 312 376 538 692 820 2,748 

2017 24 342 230 352 420 476 1,844 

2018 10 498 280 484 518 552 2,342 

Total 44 1,152 886 1,374 1,630 1,848 6,934 

Math (No Mobility Control) Cohort, Table 3B 

Grade 3 Grade4 Grade5 Grade6 Grade7 Grade 8 Total 

2016 10 314 376 538 692 820 2,750 

2017 22 342 228 352 428 484 1,856 

2018 12 500 284 484 510 550 2,340 

Total 44 1,156 888 1,374 1,630 1,854 6,946 

ELA (Mobility Control) Cohort, Table 6A 

Grade 3 Grade4. Grade5 Grade6 Grade7 Grade 8 Total 

2016 10 306 374 528 680 812 2,710 
2017 22: 342 226 350 416 470 1,826 
2018 10 488 268 480 504 538 2,288 
Total 42 1,136 868 1,358 1,600 1,820 6,824 

Math (Mobility Control) Cohort, Table 6B 

Grade 3 Grade4 Grade5 Grade6 Grade7 Grade 8 Total 

2016 10 310 372 534 678 796 2,700 

2017 22 334 222 338 410 464 1,790 

2018 12 492 268 460 506 552 2,290 

Total 44 1,136 862 1,332 1,594 1,812 6,780 
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ELA (No Transfers) Cohort, Table 8 


Grade 3 Grade4. Grade5 Grade6 Grade7 Grade 8 Total 
2016 6 218 262 366 456 586 1,894 
2017 pe) 228 132 194 178 238 992 
2018 4 250 62 82 102 148 648 
Total 32 696 456 642 736 972 3,534 


Math (No Transfers) Cohort, Table 8 


Grade 3 Grade4. Grade5 Grade6 Grade7 Grade 8 Total 
2016 6 220 260 364 452 586 1,888 
2017 20 226 130 190 172 232 970 
2018 4 250 56 80 104 154 648 
Total 30 696 446 634 728 972 3,506 


